With advances in integrated circuit, microprocessor, networking and communication technologies, increasing numbers of devices, in particular, digital computing devices, are being networked together. Devices are often first coupled to a local area network, such as an Ethernet based office/home network. In turn, local area networks are interconnected together through wide area networks, such as ATM networks, Frame Relays, and the like. Of particular interest is the TCP/IP based global inter-networks—the Internet.
As a result of this trend of increased connectivity, increasing numbers of applications that are network dependent are being deployed. Examples of these network dependent applications include but are not limited to, email, net-based telephony, world wide web and various types of e-commerce. For these applications, success inherently means a high volume of desirable network traffic for their implementing servers. To ensure continuing success, quality of service through orderly and efficient handling of the large volume of desirable network traffic has become of paramount importance. Various subject matters, such as scalability, distributive deployment and caching of contents as well as regulating network traffic destined for a network node, have become of great interest.
Unfortunately, success also may mean attracting undesirable network traffic designed to disrupt or completely shut down the services offered by the implementing servers. To ensure continuing success, the ability to fend off undesirable network traffic, also known as fending off denial of service (DoS) attacks, has also become of great importance. Various subject matters, including detection and filtering of packets with spoofed web crawler requests, have too become of great interest.
However, to-date, there are limited approaches for detecting and filtering out packets with spoofed web crawler requests. Thus it is desirable to optimize spoofed web crawler detection and filtering techniques.